Papers with text generation systems
Evaluating Factual Consistency of Texts with Semantic Role Labeling (2023.starsem-1)
Copied to clipboard
| Challenge: | Existing evaluation methods rely on task-specific language models, which in turn hampers interpretation of generated scores. |
| Approach: | They propose a reference-free evaluation metric for text summarization that measures factuality . their method generates fact tuples from Semantic Role Labels, applied to both input and summary texts. |
| Outcome: | The proposed evaluation metric is comparable with state-of-the-art methods and has a stable generalization across datasets. |
GLTR: Statistical Detection and Visualization of Generated Text (P19-3)
Copied to clipboard
| Challenge: | GLTR is a tool to detect generated text that can be used by non-experts. |
| Approach: | They propose a tool to detect generated text using a set of statistical methods that can be used by non-experts. |
| Outcome: | The proposed method improves detection rate of fake text from 54% to 72% without training. |
MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance (D19-1)
Copied to clipboard
| Challenge: | Existing evaluation metrics are not capable of evaluating text quality. |
| Approach: | They propose a metric that compares system output against reference texts based on semantics rather than surface forms. |
| Outcome: | The proposed metric shows a high correlation with human judgment of text quality on a number of text generation tasks. |
Sentence-Level Content Planning and Style Specification for Neural Text Generation (D19-1)
Copied to clipboard
| Challenge: | Recent advances in text generation systems often produce incoherent and unfaithful outputs . a novel automated text generation system takes into account content selection, text planning, and surface realization. |
| Approach: | They propose an end-to-end trained two-step text generation model that considers sentence-level content planners and language styles. |
| Outcome: | The proposed model outperforms competing models in three domains with diverse topics and varying language styles. |
Does the Order of Training Samples Matter? Improving Neural Data-to-Text Generation with Curriculum Learning (2021.eacl-main)
Copied to clipboard
| Challenge: | Recent advances in data-to-text generation have been focused on curriculum learning, which is a process of presenting training data in a specific order, starting from easy examples and moving on to more difficult ones, as the learner becomes more competent. |
| Approach: | They propose to use a curriculum learning process to change the order of training samples in a model based on the model's competence to improve model performance and convergence speed. |
| Outcome: | The proposed model shows faster convergence speed and reduced training time by 38.7% and performance by 4.84 BLEU. |
Automatic Detection of Generated Text is Easiest when Humans are Fooled (2020.acl-main)
Copied to clipboard
| Challenge: | Recent advances in neural language modelling make it possible to rapidly generate vast amounts of human-sounding text. |
| Approach: | They compare decoding methods with popular sampling-based decoding strategies . they show that multi-sentence excerpts can fool expert human raters over 30% of the time . |
| Outcome: | The proposed methods improve with longer excerpt length, but multi-sentence excerpts fool human raters over 30% of the time. |
Favi-Score: A Measure for Favoritism in Automated Preference Ratings for Generative AI Evaluation (2024.acl-long)
Copied to clipboard
| Challenge: | Generative AI systems are becoming ubiquitous for all kinds of modalities . evaluation of generated outputs is increasingly difficult due to cost and complexity of human evaluations. |
| Approach: | They propose to evaluate preference ratings on sign accuracy and favoritism . they propose to use automated metrics to assess generated outputs . |
| Outcome: | The proposed evaluations of preference ratings rely on correlation to human judgments or sign accuracy scores, but this does not tell the whole story. |
DiscoScore: Evaluating Text Generation with BERT and Discourse Coherence (2023.eacl-main)
Copied to clipboard
| Challenge: | DiscoScore is a parametrized discourse metric that uses BERT to model discourse coherence . it is weak when operated at system level, and is therefore not reliable in a way to spot improvements . |
| Approach: | They propose a parametrized discourse metric which uses BERT to model discourse coherence from different perspectives. |
| Outcome: | The proposed model outperforms existing models on document-level machine translation and summarization. |
R2D2: Robust Data-to-Text with Replacement Detection (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to mitigate unfaithful text generation are inadequate . data-to-text generation requires a structured input format . |
| Approach: | They propose a training framework that addresses unfaithful Data-to-Text generation by training a system as a generator and faithfulness discriminator with additional replacement detection and unlikelihood learning tasks. |
| Outcome: | The proposed training framework improves FeTaQA, LogicNLG, and ToTTo fidelity on D2T systems. |
Generating Diverse and High-Quality Texts by Minimum Bayes Risk Decoding (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing decoding algorithms to generate diverse outputs are based on beam search or random sampling, thus their output quality is capped by these underlying decoding methods. |
| Approach: | They propose to add a diversity penalty to MBR decoding and a clustering problem to create diversity-promoting decoding algorithms by enforcing diversity objectives. |
| Outcome: | The proposed method achieves a better trade-off than the diverse beam search and sampling algorithms overall. |
Evaluating the Morphosyntactic Well-formedness of Generated Texts (2021.emnlp-main)
Copied to clipboard
Adithya Pratapa, Antonios Anastasopoulos, Shruti Rijhwani, Aditi Chaudhary, David R. Mortensen, Graham Neubig, Yulia Tsvetkov
| Challenge: | Text generation systems are ubiquitous in natural language processing applications, but evaluation of these systems remains a challenge, especially in multilingual settings. |
| Approach: | They propose a metric to evaluate the morphosyntactic well-formedness of text using its dependency parse and morphologically-rich rules of the language. |
| Outcome: | The proposed metric can evaluate the morphosyntactic well-formedness of text using its dependency parse and morphologically-rich rules of the language. |
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors (2021.emnlp-main)
Copied to clipboard
| Challenge: | Evaluation metrics are a key ingredient for progress of text generation systems . a class of novel evaluation metrics based on BERT and its variants has been explored . |
| Approach: | They propose to disentangle BERT-based evaluation metrics along linguistic factors . they show they are sensitive to lexical overlap, just like BLEU and ROUGE . |
| Outcome: | The proposed metrics capture all aspects but are sensitive to lexical overlap, just like BLEU and ROUGE, the authors show . |